1. Technical Field
The subject matter described herein generally relates to the field of artificial intelligence and, more particularly, to systems and methods for teaching machines to mimic human processing using crowd-sourced prediction.
2. Background Information
Crowdsourcing has emerged as an effective answer to a variety of problems, ranging from the discovery of innovative solutions to open challenges in research, to the use of humans for performing tiny tasks that are easy for humans, yet remain difficult for even sophisticated algorithms. Amazon's Mechanical Turk, specifically, has proven to be an innovator in crowdsourcing technology, allowing computers to get programmatic access to human intelligence, through an API: computer programs could post micro-tasks on the Amazon Mechanical Turk market and on the other side of the API a human could complete the task and send back the answer. See https://www.mturk.com/mturk/welcome.
The introduction of such products and services gave birth to a new “crowdsourcing-based” industry, which promises to create solutions for a variety of problems that were so far too difficult to tackle using computers. Due to the extremely low costs often associated with crowdsourcing, crowdsourcing-based services have been introduced for many problem domains in which it was possible, albeit expensive, to develop automatic solutions.
The use of crowdsourcing to improve machine learning algorithms is a topic that attracted significant interest over the last few years. For example, the ReCAPTCHA project is using crowdsourcing human intelligence to recognize words in scanned documents that are not recognizable by existing OCR systems. Then the data are being used to train further and hopefully improve the existing automatic OCR system. See Luis von Ahn, Ben Maurer, Colin McMillen, David Abraham and Manuel Blum (2008), “reCAPTCHA: Human-Based Character Recognition via Web Security Measures” (PDF), Science 321 (5895): 1465-1468. Another system uses crowds to learn a human-based similarity kernel to understand what images are similar. Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai (2011), Adaptively Learning the Crowd Kernel, ICML: 9. Still another related approach is a crowdsourcing website to share cybersecurity threat information, and then use the data to learn models that detect malicious websites. Eugene Fink, Mehrbod Sharifi, and Jaime G. Carbonell (2011), Application of Machine Learning and Crowdsourcing to Detection of Cybersecurity Threats. Computer Science, Carnegie Mellon University.
Nonetheless, some attempted solutions applying blind adoption of crowdsourcing have been regressive and have failed to take advantage of decades of research in computer science. It would be advantageous if there were a system and method that consistently learned how to perform tasks from observing human behavior and, once a threshold level of performance has been reached for a particular task, inserted machine processing of that task in place of human processing.
The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the embodiments described herein.